Skip to content

Mongodb support for partial updates. - #380

Merged
adiom-mark merged 1 commit into
mainfrom
mongo_partial
Apr 17, 2026
Merged

Mongodb support for partial updates.#380
adiom-mark merged 1 commit into
mainfrom
mongo_partial

Conversation

@adiom-mark

@adiom-mark adiom-mark commented Apr 17, 2026

Copy link
Copy Markdown
Collaborator

Summary by CodeRabbit

  • Bug Fixes

    • Improved error handling for invalid update identifiers in batch operations
    • Enhanced deduplication logic to correctly handle multiple updates to the same record
  • New Features

    • Added support for ordered batch write operations
    • Improved validation and filtering of ID fields during database updates

@coderabbitai

coderabbitai Bot commented Apr 17, 2026

Copy link
Copy Markdown

Warning

Rate limit exceeded

@adiom-mark has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 43 minutes and 18 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 43 minutes and 18 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: deb0b7fa-897f-4750-a217-4bbeccc8f6a8

📥 Commits

Reviewing files that changed from the base of the PR and between 32b3fc6 and fc09988.

📒 Files selected for processing (5)
  • connectors/mongo/conn.go
  • connectors/mongo/conn_unit_test.go
  • connectors/mongo/connector_test.go
  • connectors/util/util.go
  • connectors/util/util_test.go
📝 Walkthrough

Walkthrough

This PR refactors the MongoDB connector's WriteUpdates method by replacing the previous simple approach with a new pipeline that validates ID-based filters, removes ID-targeted fields from payloads, deduplicates updates per ID, handles ordered vs. unordered bulk writes, and introduces a utility function for building stable keys from BSON ID values. Extensive unit and integration tests validate the new behavior.

Changes

Cohort / File(s) Summary
Core WriteUpdates refactor
connectors/mongo/conn.go
Replaced KeepLastUpdate logic with new pipeline: buildIdFilter to validate and derive MongoDB filters from update IDs (respecting FullDocumentKey gating), stripIdFields to remove ID-targeted $set fields, and buildBulkModels to transform updates into bulk models with per-ID deduplication and special ordering rules (e.g., delete-after-inserts collapses to delete only). Updated WriteUpdates to call buildBulkModels and pass ordered/unordered options based on composition needs.
Utility for ID key generation
connectors/util/util.go, connectors/util/util_test.go
Added BsonIdKey() function to build stable, collision-resistant keys from ordered BSON ID values using length-prefixed names, type bytes, and data payloads. Includes comprehensive test coverage for key uniqueness and collision detection.
Unit test suite
connectors/mongo/conn_unit_test.go
Comprehensive unit tests for buildIdFilter (validates empty ID detection, backward-compat unnamed ID parts, and FullDocumentKey gating), stripIdFields (validates byte-level removal and nil returns), and buildBulkModels (validates deduplication, insert/delete composition rules, partial update ordering, and $unset field filtering).
Integration test additions
connectors/mongo/connector_test.go
Added test helpers (assertDoc, assertNoDoc, toBson, toBsonID) and new TestMongoConnectorUpdates that validates end-to-end write behavior with unordered and ordered bulk writes across multiple document IDs, including insert, partial update, and delete operations.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~40 minutes

Possibly related PRs

Poem

🐰 A filter here, a field stripped there,
Bulk writes bloom with ordered care,
Dedup hops through each update's ID,
One last write remains, you'll see!

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 16.22% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title 'Mongodb support for partial updates' directly and clearly summarizes the main change: adding partial update functionality to the MongoDB connector, which is the central focus of all modifications across the codebase.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch mongo_partial

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (3)
connectors/mongo/connector_test.go (1)

68-77: Test helpers swallow/abuse Decode — small robustness fix.

  • assertDoc ignores the Decode error; if FindOne fails for a reason other than "document missing", res will be nil and the assert.Equal message will misleadingly point at a value mismatch instead of the underlying driver error.
  • assertNoDoc passes nil to Decode, which is fragile (behavior when the document unexpectedly exists depends on driver internals). Prefer .Err() to check the not-found condition directly.
♻️ Suggested change
 func assertDoc(t *testing.T, col *mongo.Collection, expected map[string]string) {
 	var res map[string]string
-	col.FindOne(t.Context(), bson.M{"_id": expected["_id"]}).Decode(&res)
+	require.NoError(t, col.FindOne(t.Context(), bson.M{"_id": expected["_id"]}).Decode(&res))
 	assert.Equal(t, expected, res)
 }

 func assertNoDoc(t *testing.T, col *mongo.Collection, id string) {
-	err := col.FindOne(t.Context(), bson.M{"_id": id}).Decode(nil)
-	assert.ErrorIs(t, err, mongo.ErrNoDocuments)
+	assert.ErrorIs(t, col.FindOne(t.Context(), bson.M{"_id": id}).Err(), mongo.ErrNoDocuments)
 }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@connectors/mongo/connector_test.go` around lines 68 - 77, assertDoc currently
ignores the error from FindOne().Decode which can hide driver errors and produce
misleading assertions; change assertDoc to capture and assert.NoError on the
Decode call (e.g., err := col.FindOne(...).Decode(&res); assert.NoError(t, err))
before asserting equality with expected, and in assertNoDoc replace the
Decode(nil) usage with checking the FindOne(...).Err() result (e.g., err :=
col.FindOne(...).Err(); assert.ErrorIs(t, err, mongo.ErrNoDocuments)) to
reliably detect the "not found" condition; update the functions named assertDoc
and assertNoDoc accordingly.
connectors/mongo/conn.go (2)

1010-1025: Duplicated idKeys map build.

stripIdFields (line 925) already constructed the same idKeys set from idFilter. Rebuilding it here is redundant; you could either hoist the set above the switch or expose a helper that returns it from stripIdFields. Minor, but removes duplication and one allocation per partial update.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@connectors/mongo/conn.go` around lines 1010 - 1025, The idKeys map is rebuilt
redundantly inside the partial-unset block; reuse the idKeys computed by
stripIdFields instead of allocating a new map. Modify stripIdFields to either
return the idKeys set alongside its current result or compute idKeys once before
the switch that handles partial updates, then reference that idKeys in the
partial-unset logic (the code using idFilter and
update.GetPartialUpdateUnset()). Remove the inner idKeys construction to
eliminate the duplicate allocation.

979-984: Dead assignment in the isNew branch.

lastType is only consulted in the !isNew branches of each case below; when isNew is true the code always appends to models regardless of lastType. The lastType = update.GetType() assignment is never read. Either drop it, or simplify by only tracking "was the last op a partial update" in seen (a map[string]bool), which also removes the adiomv1.UpdateType import-coupling on what is really a boolean signal.

♻️ Minor simplification
-	seen := map[string]adiomv1.UpdateType{}
+	lastIsPartial := map[string]bool{}

 	for i := len(updates) - 1; i >= 0; i-- {
 		update := updates[i]
 		idFilter, idKey, err := c.buildIdFilter(update)
 		if err != nil {
 			return nil, false, err
 		}

-		lastType, found := seen[idKey]
-		isNew := !found
-		if isNew {
-			seen[idKey] = update.GetType()
-			lastType = update.GetType()
-		}
+		last, found := lastIsPartial[idKey]
+		isNew := !found
+		if isNew {
+			lastIsPartial[idKey] = update.GetType() == adiomv1.UpdateType_UPDATE_TYPE_PARTIAL_UPDATE
+			last = lastIsPartial[idKey]
+		}

Then replace lastType == adiomv1.UpdateType_UPDATE_TYPE_PARTIAL_UPDATE with last.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@connectors/mongo/conn.go` around lines 979 - 984, The dead assignment sets
lastType = update.GetType() in the isNew branch but lastType is only used for
non-new paths; remove this unused assignment and simplify the seen map to track
only whether the last op was a partial update (e.g., change seen from
map[string]adiomv1.UpdateType to map[string]bool or a new seenPartial map keyed
by idKey), update the branch that sets seen[idKey] to store a boolean (true if
update.GetType() == adiomv1.UpdateType_UPDATE_TYPE_PARTIAL_UPDATE), and replace
all uses of lastType == adiomv1.UpdateType_UPDATE_TYPE_PARTIAL_UPDATE with the
boolean flag (e.g., last) so models appending logic remains correct and the
adiomv1 type coupling is removed; also drop the now-unread lastType variable and
its assignment in the isNew branch.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@connectors/mongo/conn.go`:
- Around line 924-958: stripIdFields currently swallows errors and returns the
original raw BSON, which can leave immutable _id/shard keys and cause Mongo
update failures; change stripIdFields to return (bson.Raw, error) instead of
just bson.Raw, return a non-nil error when raw.Elements() or
bson.Marshal(filtered) fails (and return nil, nil if filtering removes all
fields), and update the PARTIAL_UPDATE caller to handle the error (log/abort the
bulk op or skip the patch) rather than assigning the unchanged raw into $set;
reference the stripIdFields function and the PARTIAL_UPDATE path where its
result is used so callers can properly react to failures.

---

Nitpick comments:
In `@connectors/mongo/conn.go`:
- Around line 1010-1025: The idKeys map is rebuilt redundantly inside the
partial-unset block; reuse the idKeys computed by stripIdFields instead of
allocating a new map. Modify stripIdFields to either return the idKeys set
alongside its current result or compute idKeys once before the switch that
handles partial updates, then reference that idKeys in the partial-unset logic
(the code using idFilter and update.GetPartialUpdateUnset()). Remove the inner
idKeys construction to eliminate the duplicate allocation.
- Around line 979-984: The dead assignment sets lastType = update.GetType() in
the isNew branch but lastType is only used for non-new paths; remove this unused
assignment and simplify the seen map to track only whether the last op was a
partial update (e.g., change seen from map[string]adiomv1.UpdateType to
map[string]bool or a new seenPartial map keyed by idKey), update the branch that
sets seen[idKey] to store a boolean (true if update.GetType() ==
adiomv1.UpdateType_UPDATE_TYPE_PARTIAL_UPDATE), and replace all uses of lastType
== adiomv1.UpdateType_UPDATE_TYPE_PARTIAL_UPDATE with the boolean flag (e.g.,
last) so models appending logic remains correct and the adiomv1 type coupling is
removed; also drop the now-unread lastType variable and its assignment in the
isNew branch.

In `@connectors/mongo/connector_test.go`:
- Around line 68-77: assertDoc currently ignores the error from FindOne().Decode
which can hide driver errors and produce misleading assertions; change assertDoc
to capture and assert.NoError on the Decode call (e.g., err :=
col.FindOne(...).Decode(&res); assert.NoError(t, err)) before asserting equality
with expected, and in assertNoDoc replace the Decode(nil) usage with checking
the FindOne(...).Err() result (e.g., err := col.FindOne(...).Err();
assert.ErrorIs(t, err, mongo.ErrNoDocuments)) to reliably detect the "not found"
condition; update the functions named assertDoc and assertNoDoc accordingly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 593996a5-6f4a-4e63-a82a-bcab3a34f783

📥 Commits

Reviewing files that changed from the base of the PR and between 0981a82 and 32b3fc6.

📒 Files selected for processing (5)
  • connectors/mongo/conn.go
  • connectors/mongo/conn_unit_test.go
  • connectors/mongo/connector_test.go
  • connectors/util/util.go
  • connectors/util/util_test.go

Comment thread connectors/mongo/conn.go Outdated
@adiom-mark
adiom-mark merged commit 3ee24c0 into main Apr 17, 2026
2 checks passed
@adiom-mark
adiom-mark deleted the mongo_partial branch April 17, 2026 20:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant